{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "improving_neural_net_performance.ipynb",
      "version": "0.3.2",
      "views": {},
      "default_view": {},
      "provenance": [],
      "collapsed_sections": [
        "jFfc3saSxg6t",
        "FSPZIiYgyh93",
        "GhFtWjQRzD2l",
        "P8BLQ7T71JWd",
        "copyright-notice"
      ]
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "copyright-notice"
      },
      "source": [
        "#### Copyright 2017 Google LLC."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "copyright-notice2",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          }
        },
        "cellView": "both"
      },
      "outputs": [],
      "source": [
        "# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
        "# you may not use this file except in compliance with the License.\n",
        "# You may obtain a copy of the License at\n",
        "#\n",
        "# https://www.apache.org/licenses/LICENSE-2.0\n",
        "#\n",
        "# Unless required by applicable law or agreed to in writing, software\n",
        "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
        "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
        "# See the License for the specific language governing permissions and\n",
        "# limitations under the License."
      ],
      "execution_count": 0
    },
    {
      "metadata": {
        "id": "eV16J6oUY-HN",
        "colab_type": "text",
        "slideshow": {
          "slide_type": "slide"
        }
      },
      "cell_type": "markdown",
      "source": [
        " # Am\u00e9liorer les performances du r\u00e9seau de neurones"
      ]
    },
    {
      "metadata": {
        "id": "0Rwl1iXIKxkm",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        " **Objectif d'apprentissage\u00a0:** Am\u00e9liorer les performances d'un r\u00e9seau de neurones en normalisant des caract\u00e9ristiques et en appliquant diff\u00e9rents algorithmes d'optimisation\n",
        "\n",
        "**REMARQUE\u00a0:** Les m\u00e9thodes d'optimisation d\u00e9crites dans cet exercice ne sont pas propres aux r\u00e9seaux de neurones\u00a0; elles peuvent, en effet, servir \u00e0 am\u00e9liorer la plupart des types de mod\u00e8les."
      ]
    },
    {
      "metadata": {
        "id": "lBPTONWzKxkn",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        " ## Configuration\n",
        "\n",
        "Vous allez commencer par charger les donn\u00e9es."
      ]
    },
    {
      "metadata": {
        "id": "VtYVuONUKxko",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          }
        }
      },
      "source": [
        "from __future__ import print_function\n",
        "\n",
        "import math\n",
        "\n",
        "from IPython import display\n",
        "from matplotlib import cm\n",
        "from matplotlib import gridspec\n",
        "from matplotlib import pyplot as plt\n",
        "import numpy as np\n",
        "import pandas as pd\n",
        "from sklearn import metrics\n",
        "import tensorflow as tf\n",
        "from tensorflow.python.data import Dataset\n",
        "\n",
        "tf.logging.set_verbosity(tf.logging.ERROR)\n",
        "pd.options.display.max_rows = 10\n",
        "pd.options.display.float_format = '{:.1f}'.format\n",
        "\n",
        "california_housing_dataframe = pd.read_csv(\"https://download.mlcc.google.com/mledu-datasets/california_housing_train.csv\", sep=\",\")\n",
        "\n",
        "california_housing_dataframe = california_housing_dataframe.reindex(\n",
        "    np.random.permutation(california_housing_dataframe.index))"
      ],
      "cell_type": "code",
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "B8qC-jTIKxkr",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          }
        }
      },
      "source": [
        "def preprocess_features(california_housing_dataframe):\n",
        "  \"\"\"Prepares input features from California housing data set.\n",
        "\n",
        "  Args:\n",
        "    california_housing_dataframe: A Pandas DataFrame expected to contain data\n",
        "      from the California housing data set.\n",
        "  Returns:\n",
        "    A DataFrame that contains the features to be used for the model, including\n",
        "    synthetic features.\n",
        "  \"\"\"\n",
        "  selected_features = california_housing_dataframe[\n",
        "    [\"latitude\",\n",
        "     \"longitude\",\n",
        "     \"housing_median_age\",\n",
        "     \"total_rooms\",\n",
        "     \"total_bedrooms\",\n",
        "     \"population\",\n",
        "     \"households\",\n",
        "     \"median_income\"]]\n",
        "  processed_features = selected_features.copy()\n",
        "  # Create a synthetic feature.\n",
        "  processed_features[\"rooms_per_person\"] = (\n",
        "    california_housing_dataframe[\"total_rooms\"] /\n",
        "    california_housing_dataframe[\"population\"])\n",
        "  return processed_features\n",
        "\n",
        "def preprocess_targets(california_housing_dataframe):\n",
        "  \"\"\"Prepares target features (i.e., labels) from California housing data set.\n",
        "\n",
        "  Args:\n",
        "    california_housing_dataframe: A Pandas DataFrame expected to contain data\n",
        "      from the California housing data set.\n",
        "  Returns:\n",
        "    A DataFrame that contains the target feature.\n",
        "  \"\"\"\n",
        "  output_targets = pd.DataFrame()\n",
        "  # Scale the target to be in units of thousands of dollars.\n",
        "  output_targets[\"median_house_value\"] = (\n",
        "    california_housing_dataframe[\"median_house_value\"] / 1000.0)\n",
        "  return output_targets"
      ],
      "cell_type": "code",
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "Ah6LjMIJ2spZ",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          }
        }
      },
      "source": [
        "# Choose the first 12000 (out of 17000) examples for training.\n",
        "training_examples = preprocess_features(california_housing_dataframe.head(12000))\n",
        "training_targets = preprocess_targets(california_housing_dataframe.head(12000))\n",
        "\n",
        "# Choose the last 5000 (out of 17000) examples for validation.\n",
        "validation_examples = preprocess_features(california_housing_dataframe.tail(5000))\n",
        "validation_targets = preprocess_targets(california_housing_dataframe.tail(5000))\n",
        "\n",
        "# Double-check that we've done the right thing.\n",
        "print(\"Training examples summary:\")\n",
        "display.display(training_examples.describe())\n",
        "print(\"Validation examples summary:\")\n",
        "display.display(validation_examples.describe())\n",
        "\n",
        "print(\"Training targets summary:\")\n",
        "display.display(training_targets.describe())\n",
        "print(\"Validation targets summary:\")\n",
        "display.display(validation_targets.describe())"
      ],
      "cell_type": "code",
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "NqIbXxx222ea",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        " ## Entra\u00eener le r\u00e9seau de neurones\n",
        "\n",
        "Vous allez ensuite entra\u00eener le r\u00e9seau de neurones."
      ]
    },
    {
      "metadata": {
        "id": "6k3xYlSg27VB",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          }
        }
      },
      "source": [
        "def construct_feature_columns(input_features):\n",
        "  \"\"\"Construct the TensorFlow Feature Columns.\n",
        "\n",
        "  Args:\n",
        "    input_features: The names of the numerical input features to use.\n",
        "  Returns:\n",
        "    A set of feature columns\n",
        "  \"\"\" \n",
        "  return set([tf.feature_column.numeric_column(my_feature)\n",
        "              for my_feature in input_features])"
      ],
      "cell_type": "code",
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "De9jwyy4wTUT",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          }
        }
      },
      "source": [
        "def my_input_fn(features, targets, batch_size=1, shuffle=True, num_epochs=None):\n",
        "    \"\"\"Trains a neural network model.\n",
        "  \n",
        "    Args:\n",
        "      features: pandas DataFrame of features\n",
        "      targets: pandas DataFrame of targets\n",
        "      batch_size: Size of batches to be passed to the model\n",
        "      shuffle: True or False. Whether to shuffle the data.\n",
        "      num_epochs: Number of epochs for which data should be repeated. None = repeat indefinitely\n",
        "    Returns:\n",
        "      Tuple of (features, labels) for next data batch\n",
        "    \"\"\"\n",
        "    \n",
        "    # Convert pandas data into a dict of np arrays.\n",
        "    features = {key:np.array(value) for key,value in dict(features).items()}                                           \n",
        " \n",
        "    # Construct a dataset, and configure batching/repeating.\n",
        "    ds = Dataset.from_tensor_slices((features,targets)) # warning: 2GB limit\n",
        "    ds = ds.batch(batch_size).repeat(num_epochs)\n",
        "    \n",
        "    # Shuffle the data, if specified.\n",
        "    if shuffle:\n",
        "      ds = ds.shuffle(10000)\n",
        "    \n",
        "    # Return the next batch of data.\n",
        "    features, labels = ds.make_one_shot_iterator().get_next()\n",
        "    return features, labels"
      ],
      "cell_type": "code",
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "W-51R3yIKxk4",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          }
        }
      },
      "source": [
        "def train_nn_regression_model(\n",
        "    my_optimizer,\n",
        "    steps,\n",
        "    batch_size,\n",
        "    hidden_units,\n",
        "    training_examples,\n",
        "    training_targets,\n",
        "    validation_examples,\n",
        "    validation_targets):\n",
        "  \"\"\"Trains a neural network regression model.\n",
        "  \n",
        "  In addition to training, this function also prints training progress information,\n",
        "  as well as a plot of the training and validation loss over time.\n",
        "  \n",
        "  Args:\n",
        "    my_optimizer: An instance of `tf.train.Optimizer`, the optimizer to use.\n",
        "    steps: A non-zero `int`, the total number of training steps. A training step\n",
        "      consists of a forward and backward pass using a single batch.\n",
        "    batch_size: A non-zero `int`, the batch size.\n",
        "    hidden_units: A `list` of int values, specifying the number of neurons in each layer.\n",
        "    training_examples: A `DataFrame` containing one or more columns from\n",
        "      `california_housing_dataframe` to use as input features for training.\n",
        "    training_targets: A `DataFrame` containing exactly one column from\n",
        "      `california_housing_dataframe` to use as target for training.\n",
        "    validation_examples: A `DataFrame` containing one or more columns from\n",
        "      `california_housing_dataframe` to use as input features for validation.\n",
        "    validation_targets: A `DataFrame` containing exactly one column from\n",
        "      `california_housing_dataframe` to use as target for validation.\n",
        "      \n",
        "  Returns:\n",
        "    A tuple `(estimator, training_losses, validation_losses)`:\n",
        "      estimator: the trained `DNNRegressor` object.\n",
        "      training_losses: a `list` containing the training loss values taken during training.\n",
        "      validation_losses: a `list` containing the validation loss values taken during training.\n",
        "  \"\"\"\n",
        "\n",
        "  periods = 10\n",
        "  steps_per_period = steps / periods\n",
        "  \n",
        "  # Create a DNNRegressor object.\n",
        "  my_optimizer = tf.contrib.estimator.clip_gradients_by_norm(my_optimizer, 5.0)\n",
        "  dnn_regressor = tf.estimator.DNNRegressor(\n",
        "      feature_columns=construct_feature_columns(training_examples),\n",
        "      hidden_units=hidden_units,\n",
        "      optimizer=my_optimizer\n",
        "  )\n",
        "  \n",
        "  # Create input functions.\n",
        "  training_input_fn = lambda: my_input_fn(training_examples, \n",
        "                                          training_targets[\"median_house_value\"], \n",
        "                                          batch_size=batch_size)\n",
        "  predict_training_input_fn = lambda: my_input_fn(training_examples, \n",
        "                                                  training_targets[\"median_house_value\"], \n",
        "                                                  num_epochs=1, \n",
        "                                                  shuffle=False)\n",
        "  predict_validation_input_fn = lambda: my_input_fn(validation_examples, \n",
        "                                                    validation_targets[\"median_house_value\"], \n",
        "                                                    num_epochs=1, \n",
        "                                                    shuffle=False)\n",
        "\n",
        "  # Train the model, but do so inside a loop so that we can periodically assess\n",
        "  # loss metrics.\n",
        "  print(\"Training model...\")\n",
        "  print(\"RMSE (on training data):\")\n",
        "  training_rmse = []\n",
        "  validation_rmse = []\n",
        "  for period in range (0, periods):\n",
        "    # Train the model, starting from the prior state.\n",
        "    dnn_regressor.train(\n",
        "        input_fn=training_input_fn,\n",
        "        steps=steps_per_period\n",
        "    )\n",
        "    # Take a break and compute predictions.\n",
        "    training_predictions = dnn_regressor.predict(input_fn=predict_training_input_fn)\n",
        "    training_predictions = np.array([item['predictions'][0] for item in training_predictions])\n",
        "    \n",
        "    validation_predictions = dnn_regressor.predict(input_fn=predict_validation_input_fn)\n",
        "    validation_predictions = np.array([item['predictions'][0] for item in validation_predictions])\n",
        "    \n",
        "    # Compute training and validation loss.\n",
        "    training_root_mean_squared_error = math.sqrt(\n",
        "        metrics.mean_squared_error(training_predictions, training_targets))\n",
        "    validation_root_mean_squared_error = math.sqrt(\n",
        "        metrics.mean_squared_error(validation_predictions, validation_targets))\n",
        "    # Occasionally print the current loss.\n",
        "    print(\"  period %02d : %0.2f\" % (period, training_root_mean_squared_error))\n",
        "    # Add the loss metrics from this period to our list.\n",
        "    training_rmse.append(training_root_mean_squared_error)\n",
        "    validation_rmse.append(validation_root_mean_squared_error)\n",
        "  print(\"Model training finished.\")\n",
        "\n",
        "  # Output a graph of loss metrics over periods.\n",
        "  plt.ylabel(\"RMSE\")\n",
        "  plt.xlabel(\"Periods\")\n",
        "  plt.title(\"Root Mean Squared Error vs. Periods\")\n",
        "  plt.tight_layout()\n",
        "  plt.plot(training_rmse, label=\"training\")\n",
        "  plt.plot(validation_rmse, label=\"validation\")\n",
        "  plt.legend()\n",
        "\n",
        "  print(\"Final RMSE (on training data):   %0.2f\" % training_root_mean_squared_error)\n",
        "  print(\"Final RMSE (on validation data): %0.2f\" % validation_root_mean_squared_error)\n",
        "\n",
        "  return dnn_regressor, training_rmse, validation_rmse"
      ],
      "cell_type": "code",
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "KueReMZ9Kxk7",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          }
        }
      },
      "source": [
        "_ = train_nn_regression_model(\n",
        "    my_optimizer=tf.train.GradientDescentOptimizer(learning_rate=0.0007),\n",
        "    steps=5000,\n",
        "    batch_size=70,\n",
        "    hidden_units=[10, 10],\n",
        "    training_examples=training_examples,\n",
        "    training_targets=training_targets,\n",
        "    validation_examples=validation_examples,\n",
        "    validation_targets=validation_targets)"
      ],
      "cell_type": "code",
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "flxmFt0KKxk9",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        " ## Mise \u00e0 l'\u00e9chelle lin\u00e9aire\n",
        "Il peut \u00eatre judicieux de normaliser les entr\u00e9es pour qu'elles soient comprises entre -1 et 1. Cela permet d'\u00e9viter au SGD de s\u00e9lectionner des mesures qui sont trop grandes dans une dimension ou trop petites dans une autre. Dans le cas pr\u00e9sent, les partisans de l'optimisation num\u00e9rique remarqueront peut-\u00eatre le lien avec l'utilisation d'un pr\u00e9conditionneur."
      ]
    },
    {
      "metadata": {
        "id": "Dws5rIQjKxk-",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          }
        }
      },
      "source": [
        "def linear_scale(series):\n",
        "  min_val = series.min()\n",
        "  max_val = series.max()\n",
        "  scale = (max_val - min_val) / 2.0\n",
        "  return series.apply(lambda x:((x - min_val) / scale) - 1.0)"
      ],
      "cell_type": "code",
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "MVmuHI76N2Sz",
        "colab_type": "text",
        "slideshow": {
          "slide_type": "slide"
        }
      },
      "cell_type": "markdown",
      "source": [
        " ## T\u00e2che\u00a01\u00a0: Utiliser la mise \u00e0 l'\u00e9chelle lin\u00e9aire pour normaliser les caract\u00e9ristiques\n",
        "\n",
        "**Normalisez les entr\u00e9es sur l'\u00e9chelle -1, 1.**\n",
        "\n",
        "**Consacrez environ cinq minutes \u00e0 l'apprentissage et \u00e0 l'\u00e9valuation des derni\u00e8res donn\u00e9es normalis\u00e9es. Comment vous en sortez-vous\u00a0?**\n",
        "\n",
        "En r\u00e8gle g\u00e9n\u00e9rale, l'apprentissage des r\u00e9seaux de neurones est plus performant lorsque les caract\u00e9ristiques d'entr\u00e9e se situent grosso\u00a0modo sur la m\u00eame \u00e9chelle.\n",
        "\n",
        "\u00c9valuez l'int\u00e9grit\u00e9 de vos donn\u00e9es normalis\u00e9es. (Que se passerait-il si vous oubliiez de normaliser une caract\u00e9ristique\u00a0?)\n"
      ]
    },
    {
      "metadata": {
        "id": "yD948ZgAM6Cx",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          }
        }
      },
      "source": [
        "def normalize_linear_scale(examples_dataframe):\n",
        "  \"\"\"Returns a version of the input `DataFrame` that has all its features normalized linearly.\"\"\"\n",
        "  #\n",
        "  # Your code here: normalize the inputs.\n",
        "  #\n",
        "  pass\n",
        "\n",
        "normalized_dataframe = normalize_linear_scale(preprocess_features(california_housing_dataframe))\n",
        "normalized_training_examples = normalized_dataframe.head(12000)\n",
        "normalized_validation_examples = normalized_dataframe.tail(5000)\n",
        "\n",
        "_ = train_nn_regression_model(\n",
        "    my_optimizer=tf.train.GradientDescentOptimizer(learning_rate=0.0007),\n",
        "    steps=5000,\n",
        "    batch_size=70,\n",
        "    hidden_units=[10, 10],\n",
        "    training_examples=normalized_training_examples,\n",
        "    training_targets=training_targets,\n",
        "    validation_examples=normalized_validation_examples,\n",
        "    validation_targets=validation_targets)"
      ],
      "cell_type": "code",
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "jFfc3saSxg6t",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        " ### Solution\n",
        "\n",
        "Cliquez ci-dessous pour afficher une solution."
      ]
    },
    {
      "metadata": {
        "id": "Ax_IIQVRx4gr",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        " \u00c9tant donn\u00e9 que les valeurs minimale et maximale sont utilis\u00e9es dans le cadre de la normalisation, vous devez vous assurer que cette op\u00e9ration est effectu\u00e9e sur l'int\u00e9gralit\u00e9 de l'ensemble de donn\u00e9es en une seule fois. \n",
        "\n",
        "Cela est possible ici, car toutes les donn\u00e9es se trouvent dans un seul DataFrame. S'il y avait plusieurs ensembles de donn\u00e9es, une bonne pratique consisterait \u00e0 d\u00e9river les param\u00e8tres de normalisation de l'ensemble d'apprentissage et \u00e0 les appliquer de mani\u00e8re identique \u00e0 l'ensemble d'\u00e9valuation."
      ]
    },
    {
      "metadata": {
        "id": "D-bJBXrJx-U_",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          }
        }
      },
      "source": [
        "def normalize_linear_scale(examples_dataframe):\n",
        "  \"\"\"Returns a version of the input `DataFrame` that has all its features normalized linearly.\"\"\"\n",
        "  processed_features = pd.DataFrame()\n",
        "  processed_features[\"latitude\"] = linear_scale(examples_dataframe[\"latitude\"])\n",
        "  processed_features[\"longitude\"] = linear_scale(examples_dataframe[\"longitude\"])\n",
        "  processed_features[\"housing_median_age\"] = linear_scale(examples_dataframe[\"housing_median_age\"])\n",
        "  processed_features[\"total_rooms\"] = linear_scale(examples_dataframe[\"total_rooms\"])\n",
        "  processed_features[\"total_bedrooms\"] = linear_scale(examples_dataframe[\"total_bedrooms\"])\n",
        "  processed_features[\"population\"] = linear_scale(examples_dataframe[\"population\"])\n",
        "  processed_features[\"households\"] = linear_scale(examples_dataframe[\"households\"])\n",
        "  processed_features[\"median_income\"] = linear_scale(examples_dataframe[\"median_income\"])\n",
        "  processed_features[\"rooms_per_person\"] = linear_scale(examples_dataframe[\"rooms_per_person\"])\n",
        "  return processed_features\n",
        "\n",
        "normalized_dataframe = normalize_linear_scale(preprocess_features(california_housing_dataframe))\n",
        "normalized_training_examples = normalized_dataframe.head(12000)\n",
        "normalized_validation_examples = normalized_dataframe.tail(5000)\n",
        "\n",
        "_ = train_nn_regression_model(\n",
        "    my_optimizer=tf.train.GradientDescentOptimizer(learning_rate=0.005),\n",
        "    steps=2000,\n",
        "    batch_size=50,\n",
        "    hidden_units=[10, 10],\n",
        "    training_examples=normalized_training_examples,\n",
        "    training_targets=training_targets,\n",
        "    validation_examples=normalized_validation_examples,\n",
        "    validation_targets=validation_targets)"
      ],
      "cell_type": "code",
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "MrwtdStNJ6ZQ",
        "colab_type": "text",
        "slideshow": {
          "slide_type": "slide"
        }
      },
      "cell_type": "markdown",
      "source": [
        " ## T\u00e2cher\u00a02\u00a0: Essayer un autre optimiseur\n",
        "\n",
        "** Utilisez les optimiseurs AdaGrad et Adam, et comparez leurs performances.**\n",
        "\n",
        "L'optimiseur AdaGrad constitue une solution. L'id\u00e9e principale est que cet optimiseur adapte le taux d'apprentissage pour chaque coefficient d'un mod\u00e8le, en r\u00e9duisant, de fa\u00e7on monotone, le taux d'apprentissage effectif. Cette m\u00e9thode fonctionne parfaitement pour les probl\u00e8mes convexes, mais elle ne s'av\u00e8re pas toujours id\u00e9ale pour le probl\u00e8me non\u00a0convexe d'entra\u00eenement des r\u00e9seaux de neurones. Vous pouvez utiliser AdaGrad en indiquant `AdagradOptimizer` au lieu de `GradientDescentOptimizer`. Notez qu'un taux d'apprentissage plus \u00e9lev\u00e9 peut \u00eatre n\u00e9cessaire avec AdaGrad.\n",
        "\n",
        "Pour les probl\u00e8mes d'optimisation non\u00a0convexes, Adam se r\u00e9v\u00e8le parfois plus efficace. Pour utiliser Adam, invoquez la m\u00e9thode `tf.train.AdamOptimizer`. Cette m\u00e9thode utilise plusieurs hyperparam\u00e8tres facultatifs en tant qu'arguments, mais notre solution n'en sp\u00e9cifie qu'un seul (`learning_rate`). Dans un environnement de production, vous devez \u00eatre prudent lorsque vous sp\u00e9cifiez et r\u00e9glez les hyperparam\u00e8tres facultatifs."
      ]
    },
    {
      "metadata": {
        "id": "61GSlDvF7-7q",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          }
        }
      },
      "source": [
        "#\n",
        "# YOUR CODE HERE: Retrain the network using Adagrad and then Adam.\n",
        "#"
      ],
      "cell_type": "code",
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "FSPZIiYgyh93",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        " ### Solution\n",
        "\n",
        "Cliquez ci-dessous pour afficher la solution."
      ]
    },
    {
      "metadata": {
        "id": "X1QcIeiKyni4",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        " Essayez tout d'abord l'algorithme AdaGrad."
      ]
    },
    {
      "metadata": {
        "id": "Ntn4jJxnypGZ",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          }
        }
      },
      "source": [
        "_, adagrad_training_losses, adagrad_validation_losses = train_nn_regression_model(\n",
        "    my_optimizer=tf.train.AdagradOptimizer(learning_rate=0.5),\n",
        "    steps=500,\n",
        "    batch_size=100,\n",
        "    hidden_units=[10, 10],\n",
        "    training_examples=normalized_training_examples,\n",
        "    training_targets=training_targets,\n",
        "    validation_examples=normalized_validation_examples,\n",
        "    validation_targets=validation_targets)"
      ],
      "cell_type": "code",
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "5JUsCdRRyso3",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        " Essayez maintenant Adam."
      ]
    },
    {
      "metadata": {
        "id": "lZB8k0upyuY8",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          }
        }
      },
      "source": [
        "_, adam_training_losses, adam_validation_losses = train_nn_regression_model(\n",
        "    my_optimizer=tf.train.AdamOptimizer(learning_rate=0.009),\n",
        "    steps=500,\n",
        "    batch_size=100,\n",
        "    hidden_units=[10, 10],\n",
        "    training_examples=normalized_training_examples,\n",
        "    training_targets=training_targets,\n",
        "    validation_examples=normalized_validation_examples,\n",
        "    validation_targets=validation_targets)"
      ],
      "cell_type": "code",
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "twYgC8FGyxm6",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        " Imprimez un graphique qui affiche les mesures de perte c\u00f4te \u00e0 c\u00f4te."
      ]
    },
    {
      "metadata": {
        "id": "8RHIUEfqyzW0",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          }
        }
      },
      "source": [
        "plt.ylabel(\"RMSE\")\n",
        "plt.xlabel(\"Periods\")\n",
        "plt.title(\"Root Mean Squared Error vs. Periods\")\n",
        "plt.plot(adagrad_training_losses, label='Adagrad training')\n",
        "plt.plot(adagrad_validation_losses, label='Adagrad validation')\n",
        "plt.plot(adam_training_losses, label='Adam training')\n",
        "plt.plot(adam_validation_losses, label='Adam validation')\n",
        "_ = plt.legend()"
      ],
      "cell_type": "code",
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "UySPl7CAQ28C",
        "colab_type": "text",
        "slideshow": {
          "slide_type": "slide"
        }
      },
      "cell_type": "markdown",
      "source": [
        " ## T\u00e2che\u00a03\u00a0: Explorer d'autres m\u00e9thodes de normalisation\n",
        "\n",
        "**Essayez d'autres normalisations pour diff\u00e9rentes caract\u00e9ristiques afin d'optimiser les performances.**\n",
        "\n",
        "Si vous observez attentivement les statistiques r\u00e9capitulatives de vos donn\u00e9es transform\u00e9es, vous remarquerez peut-\u00eatre que la mise \u00e0 l'\u00e9chelle lin\u00e9aire de certaines caract\u00e9ristiques les laisse regroup\u00e9es dans le voisinage de `-1`.\n",
        "\n",
        "Par exemple, de nombreuses caract\u00e9ristiques ont une m\u00e9diane d'\u00e0 peu pr\u00e8s `-0.8`, plut\u00f4t que `0.0`."
      ]
    },
    {
      "metadata": {
        "id": "QWmm_6CGKxlH",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          }
        }
      },
      "source": [
        "_ = training_examples.hist(bins=20, figsize=(18, 12), xlabelsize=2)"
      ],
      "cell_type": "code",
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "Xx9jgEMHKxlJ",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        " Il est possible d'obtenir un meilleur r\u00e9sultat en choisissant d'autres m\u00e9thodes pour transformer ces caract\u00e9ristiques.\n",
        "\n",
        "Une mise \u00e0 l'\u00e9chelle logarithmique, par exemple, peut \u00eatre int\u00e9ressante pour certaines caract\u00e9ristiques. Quant \u00e0 l'\u00e9limination des valeurs extr\u00eames, elle permettra peut-\u00eatre de rendre plus instructif le reste de l'\u00e9chelle."
      ]
    },
    {
      "metadata": {
        "id": "baKZa6MEKxlK",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          }
        }
      },
      "source": [
        "def log_normalize(series):\n",
        "  return series.apply(lambda x:math.log(x+1.0))\n",
        "\n",
        "def clip(series, clip_to_min, clip_to_max):\n",
        "  return series.apply(lambda x:(\n",
        "    min(max(x, clip_to_min), clip_to_max)))\n",
        "\n",
        "def z_score_normalize(series):\n",
        "  mean = series.mean()\n",
        "  std_dv = series.std()\n",
        "  return series.apply(lambda x:(x - mean) / std_dv)\n",
        "\n",
        "def binary_threshold(series, threshold):\n",
        "  return series.apply(lambda x:(1 if x > threshold else 0))"
      ],
      "cell_type": "code",
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "-wCCq_ClKxlO",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        " Le bloc ci-dessus contient quelques fonctions de normalisation suppl\u00e9mentaires. Essayez-en quelques-unes ou ajoutez les v\u00f4tres.\n",
        "\n",
        "Notez que si vous normalisez la cible, vous devrez annuler la normalisation des pr\u00e9dictions pour que les mesures de perte soient comparables."
      ]
    },
    {
      "metadata": {
        "id": "8ToG-mLfMO9P",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          }
        }
      },
      "source": [
        "def normalize(examples_dataframe):\n",
        "  \"\"\"Returns a version of the input `DataFrame` that has all its features normalized.\"\"\"\n",
        "  #\n",
        "  # YOUR CODE HERE: Normalize the inputs.\n",
        "  #\n",
        "  pass\n",
        "\n",
        "normalized_dataframe = normalize(preprocess_features(california_housing_dataframe))\n",
        "normalized_training_examples = normalized_dataframe.head(12000)\n",
        "normalized_validation_examples = normalized_dataframe.tail(5000)\n",
        "\n",
        "_ = train_nn_regression_model(\n",
        "    my_optimizer=tf.train.GradientDescentOptimizer(learning_rate=0.0007),\n",
        "    steps=5000,\n",
        "    batch_size=70,\n",
        "    hidden_units=[10, 10],\n",
        "    training_examples=normalized_training_examples,\n",
        "    training_targets=training_targets,\n",
        "    validation_examples=normalized_validation_examples,\n",
        "    validation_targets=validation_targets)"
      ],
      "cell_type": "code",
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "GhFtWjQRzD2l",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        " ### Solution\n",
        "\n",
        "Cliquez ci-dessous pour afficher une solution."
      ]
    },
    {
      "metadata": {
        "id": "OMoIsUMmzK9b",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        " Il ne s'agit l\u00e0 que d'un bref aper\u00e7u des m\u00e9thodes qui peuvent \u00eatre utilis\u00e9es pour prendre en compte les donn\u00e9es. D'autres transformations peuvent se r\u00e9v\u00e9ler encore plus efficaces\u00a0!\n",
        "\n",
        "Les valeurs `households`, `median_income` et `total_bedrooms` apparaissent toutes comme \u00e9tant normalement distribu\u00e9es dans un espace logarithmique.\n",
        "\n",
        "Dans le cas des valeurs `latitude`, `longitude` et `housing_median_age`, il serait sans doute pr\u00e9f\u00e9rable qu'elles soient mises \u00e0 l'\u00e9chelle de fa\u00e7on lin\u00e9aire, comme pr\u00e9c\u00e9demment.\n",
        "\n",
        "`population`, `totalRooms` et `rooms_per_person` pr\u00e9sentent quelques anomalies extr\u00eames. Elles semblent trop extr\u00eames pour que la normalisation logarithmique ne soit d'une quelconque utilit\u00e9. Il est donc pr\u00e9f\u00e9rable de les \u00e9liminer."
      ]
    },
    {
      "metadata": {
        "id": "XDEYkPquzYCH",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          }
        }
      },
      "source": [
        "def normalize(examples_dataframe):\n",
        "  \"\"\"Returns a version of the input `DataFrame` that has all its features normalized.\"\"\"\n",
        "  processed_features = pd.DataFrame()\n",
        "\n",
        "  processed_features[\"households\"] = log_normalize(examples_dataframe[\"households\"])\n",
        "  processed_features[\"median_income\"] = log_normalize(examples_dataframe[\"median_income\"])\n",
        "  processed_features[\"total_bedrooms\"] = log_normalize(examples_dataframe[\"total_bedrooms\"])\n",
        "  \n",
        "  processed_features[\"latitude\"] = linear_scale(examples_dataframe[\"latitude\"])\n",
        "  processed_features[\"longitude\"] = linear_scale(examples_dataframe[\"longitude\"])\n",
        "  processed_features[\"housing_median_age\"] = linear_scale(examples_dataframe[\"housing_median_age\"])\n",
        "\n",
        "  processed_features[\"population\"] = linear_scale(clip(examples_dataframe[\"population\"], 0, 5000))\n",
        "  processed_features[\"rooms_per_person\"] = linear_scale(clip(examples_dataframe[\"rooms_per_person\"], 0, 5))\n",
        "  processed_features[\"total_rooms\"] = linear_scale(clip(examples_dataframe[\"total_rooms\"], 0, 10000))\n",
        "\n",
        "  return processed_features\n",
        "\n",
        "normalized_dataframe = normalize(preprocess_features(california_housing_dataframe))\n",
        "normalized_training_examples = normalized_dataframe.head(12000)\n",
        "normalized_validation_examples = normalized_dataframe.tail(5000)\n",
        "\n",
        "_ = train_nn_regression_model(\n",
        "    my_optimizer=tf.train.AdagradOptimizer(learning_rate=0.15),\n",
        "    steps=1000,\n",
        "    batch_size=50,\n",
        "    hidden_units=[10, 10],\n",
        "    training_examples=normalized_training_examples,\n",
        "    training_targets=training_targets,\n",
        "    validation_examples=normalized_validation_examples,\n",
        "    validation_targets=validation_targets)"
      ],
      "cell_type": "code",
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "b7atJTbzU9Ca",
        "colab_type": "text",
        "slideshow": {
          "slide_type": "slide"
        }
      },
      "cell_type": "markdown",
      "source": [
        " ## Exercice facultatif\u00a0: Utiliser uniquement les caract\u00e9ristiques Latitude et Longitude\n",
        "\n",
        "**Entra\u00eenez un mod\u00e8le de r\u00e9seau de neurones qui utilise seulement la latitude et la longitude comme caract\u00e9ristiques.**\n",
        "\n",
        "Les agents immobiliers ont coutume de dire que l'emplacement est la seule caract\u00e9ristique importante dans le prix d'un logement.\n",
        "Voyons si cela se confirme en entra\u00eenant un mod\u00e8le qui utilise uniquement la latitude et la longitude comme caract\u00e9ristiques.\n",
        "\n",
        "Cela ne fonctionnera correctement que si le r\u00e9seau de neurones peut apprendre des non-lin\u00e9arit\u00e9s complexes \u00e0 partir de la latitude et de la longitude.\n",
        "\n",
        "**REMARQUE\u00a0:** Vous aurez peut-\u00eatre besoin d'une structure de r\u00e9seau comportant davantage de couches que ce qui \u00e9tait n\u00e9cessaire pr\u00e9c\u00e9demment."
      ]
    },
    {
      "metadata": {
        "id": "T5McjahpamOc",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          }
        }
      },
      "source": [
        "#\n",
        "# YOUR CODE HERE: Train the network using only latitude and longitude\n",
        "#"
      ],
      "cell_type": "code",
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "P8BLQ7T71JWd",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        " ### Solution\n",
        "\n",
        "Cliquez ci-dessous pour afficher une solution."
      ]
    },
    {
      "metadata": {
        "id": "1hwaFCE71OPZ",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        " Il est conseill\u00e9 de faire en sorte que la latitude et la longitude restent normalis\u00e9es\u00a0:"
      ]
    },
    {
      "metadata": {
        "id": "djKtt4mz1ZEc",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          }
        }
      },
      "source": [
        "def location_location_location(examples_dataframe):\n",
        "  \"\"\"Returns a version of the input `DataFrame` that keeps only the latitude and longitude.\"\"\"\n",
        "  processed_features = pd.DataFrame()\n",
        "  processed_features[\"latitude\"] = linear_scale(examples_dataframe[\"latitude\"])\n",
        "  processed_features[\"longitude\"] = linear_scale(examples_dataframe[\"longitude\"])\n",
        "  return processed_features\n",
        "\n",
        "lll_dataframe = location_location_location(preprocess_features(california_housing_dataframe))\n",
        "lll_training_examples = lll_dataframe.head(12000)\n",
        "lll_validation_examples = lll_dataframe.tail(5000)\n",
        "\n",
        "_ = train_nn_regression_model(\n",
        "    my_optimizer=tf.train.AdagradOptimizer(learning_rate=0.05),\n",
        "    steps=500,\n",
        "    batch_size=50,\n",
        "    hidden_units=[10, 10, 5, 5, 5],\n",
        "    training_examples=lll_training_examples,\n",
        "    training_targets=training_targets,\n",
        "    validation_examples=lll_validation_examples,\n",
        "    validation_targets=validation_targets)"
      ],
      "cell_type": "code",
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "Dw2Mr9JZ1cRi",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        " C'est acceptable lorsqu'il n'y a que deux caract\u00e9ristiques. Toutefois, il va sans dire que les prix des propri\u00e9t\u00e9s peuvent varier sensiblement sur de courtes distances."
      ]
    }
  ]
}
